<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML
><HEAD
><TITLE
>Specifying WEB space to be indexed </TITLE
><META
NAME="GENERATOR"
CONTENT="Modular DocBook HTML Stylesheet Version 1.79"><LINK
REL="HOME"
TITLE="DataparkSearch Engine 4.51"
HREF="index.en.html"><LINK
REL="UP"
TITLE="Indexing"
HREF="dpsearch-indexing.en.html"><LINK
REL="PREVIOUS"
TITLE="Clones"
HREF="dpsearch-clones.en.html"><LINK
REL="NEXT"
TITLE="Aliases"
HREF="dpsearch-aliases.en.html"><LINK
REL="STYLESHEET"
TYPE="text/css"
HREF="datapark.css"><META
NAME="Description"
CONTENT="DataparkSearch - Full Featured Web site Open Source Search Engine Software over the Internet and Intranet Web Sites Based on SQL Database. It is a Free search software covered by GNU license."><META
NAME="Keywords"
CONTENT="shareware, freeware, download, internet, unix, utilities, search engine, text retrieval, knowledge retrieval, text search, information retrieval, database search, mining, intranet, webserver, index, spider, filesearch, meta, free, open source, full-text, udmsearch, website, find, opensource, search, searching, software, udmsearch, engine, indexing, system, web, ftp, http, cgi, php, SQL, MySQL, database, php3, FreeBSD, Linux, Unix, DataparkSearch, MacOS X, Mac OS X, Windows, 2000, NT, 95, 98, GNU, GPL, url, grabbing"></HEAD
><BODY
CLASS="sect1"
BGCOLOR="#FFFFFF"
TEXT="#000000"
LINK="#0000C4"
VLINK="#1200B2"
ALINK="#C40000"
><!--#include virtual="body-before.html"--><DIV
CLASS="NAVHEADER"
><TABLE
SUMMARY="Header navigation table"
WIDTH="100%"
BORDER="0"
CELLPADDING="0"
CELLSPACING="0"
><TR
><TH
COLSPAN="3"
ALIGN="center"
>DataparkSearch Engine 4.51: Reference manual</TH
></TR
><TR
><TD
WIDTH="10%"
ALIGN="left"
VALIGN="bottom"
><A
HREF="dpsearch-clones.en.html"
ACCESSKEY="P"
>Prev</A
></TD
><TD
WIDTH="80%"
ALIGN="center"
VALIGN="bottom"
>Chapter 3. Indexing</TD
><TD
WIDTH="10%"
ALIGN="right"
VALIGN="bottom"
><A
HREF="dpsearch-aliases.en.html"
ACCESSKEY="N"
>Next</A
></TD
></TR
></TABLE
><HR
ALIGN="LEFT"
WIDTH="100%"></DIV
><DIV
CLASS="sect1"
><H1
CLASS="sect1"
><A
NAME="follow"
>3.6. Specifying WEB space to be indexed</A
></H1
>
	
	<P
>When indexer tries to insert a new URL into database or
is trying to index an existing one, it first of all checks whether
this URL has corresponding <B
CLASS="command"
>Server</B
>, <B
CLASS="command"
>Realm</B
> or <B
CLASS="command"
>Subnet</B
> command given in
<TT
CLASS="filename"
>indexer.conf</TT
>. URLs without corresponding <B
CLASS="command"
>Server</B
>, <B
CLASS="command"
>Realm</B
>
or <B
CLASS="command"
>Subnet</B
> command are not indexed. By default those URLs which are
already in database and have no Server/Realm/Subnet commands will be deleted
from database. It may happen for example after removing some
Server/Realm/Subnet commands from <TT
CLASS="filename"
>indexer.conf</TT
>.</P
>


<P
>These commands have following format:
<PRE
CLASS="programlisting"
>[Server | Realm | Subnet] [method] [subsection] [CaseType] [MatchType] [CmpType] pattern [alias]</PRE
>
</P
>

<P
>Mandatory parameter <CODE
CLASS="option"
>pattern</CODE
> specify an URL, or it part, or pattern to compare.
</P
>




		<P
>&#13;			Optional parameter <CODE
CLASS="option"
>method</CODE
> specify an document action for this command.
May take values:
			<TT
CLASS="literal"
>Allow</TT
>, <TT
CLASS="literal"
>Disallow</TT
>, <TT
CLASS="literal"
>HrefOnly</TT
>, 
			<TT
CLASS="literal"
>CheckOnly</TT
>, <TT
CLASS="literal"
>Skip</TT
>, 	<TT
CLASS="literal"
>CheckMP3</TT
>,
			<TT
CLASS="literal"
>CheckMP3Only</TT
>. By default, the value <TT
CLASS="literal"
>Allow</TT
> is used.
		<P
></P
><OL
TYPE="1"
><LI
>&#13;				<P
>&#13;					<B
CLASS="command"
>Allow</B
>
<A
NAME="AEN809"
></A
>
				</P
>
				<P
>&#13;					Value <TT
CLASS="literal"
>Allow</TT
> specify that all corresponding documents will be indexed
 and scanned for new links. Depends on <TT
CLASS="literal"
>Content-Type</TT
> appropriate external parser is executed if need.
				</P
>
			</LI
><LI
>&#13;				<P
>&#13;					<B
CLASS="command"
>Disallow</B
>
<A
NAME="AEN819"
></A
>
				</P
>
				<P
>&#13;					Value <TT
CLASS="literal"
>Disallow</TT
> specify that all corresponding documents will be ignored and
deleted from database, if its was placed into before.
				</P
>
			</LI
><LI
>&#13;				<P
>&#13;					<B
CLASS="command"
>HrefOnly</B
>
<A
NAME="AEN828"
></A
>
				</P
>
				<P
>&#13;					Value <TT
CLASS="literal"
>HrefOnly</TT
> specify that all corresponding documents will be only
scanned for new links (not indexed). This is useful, for example, for mail archives indexing, when index pages is only scanned to
detect new messages for indexing. 
<PRE
CLASS="programlisting"
>&#13;Server HrefOnly Page http://www.mail-archive.com/general%40mnogosearch.org/
Server Allow    Path http://www.mail-archive.com/general%40mnogosearch.org/
</PRE
>
				</P
>
			</LI
><LI
>&#13;				<P
>&#13;					<B
CLASS="command"
>CheckOnly</B
>
<A
NAME="AEN838"
></A
>
				</P
>
				<P
>&#13;					Value <TT
CLASS="literal"
>CheckOnly</TT
> specify that all corresponding documents will be
requested by HTTP HEAD request, not HTTP GET, i.e. inly brief info about documents (size, last modified, content type) will be
 fetched. This allow, for example, check links on your site:
<PRE
CLASS="programlisting"
>&#13;Server HrefOnly  http://www.mnogosearch.org/
Realm  CheckOnly *
</PRE
>
</P
>
<P
>&#13;These commands instruct <B
CLASS="command"
>indexer</B
> to scan all documents on <TT
CLASS="literal"
>www.mnogosearch.org</TT
> site and
collect all links. Brief info about every document found will be requested by HEAD method.
After indexing done, <B
CLASS="command"
>indexer -S</B
> command will show status for all documents from this site.
				</P
>
			</LI
><LI
>&#13;				<P
>&#13;					<B
CLASS="command"
>Skip</B
>
<A
NAME="AEN852"
></A
>
				</P
>
				<P
>&#13;					Value <TT
CLASS="literal"
>Skip</TT
> specify that all corresponding documents will be skipped
while indexing. This is useful when need temporally disable reindexing several sites, but able search on.
These documents will marked as expired.
				</P
>
			</LI
><LI
>&#13;				<P
>&#13;					<B
CLASS="command"
>CheckMP3</B
>
<A
NAME="AEN861"
></A
>
				</P
>
				<P
>&#13;					Value <TT
CLASS="literal"
>CheckMP3</TT
> specify that corresponding documents will be checked
for MP3 tags along if its <TT
CLASS="literal"
>Content-Type</TT
> is equal to <TT
CLASS="literal"
>audio/mpeg</TT
>.
This is useful, for example, if remote server supply <TT
CLASS="literal"
>application/octet-stream</TT
> as 
<TT
CLASS="literal"
>Content-Type</TT
> for MP3 files. If this tag is present, these files will indexed as MP3 file, otherwise its
will be processed according to <TT
CLASS="literal"
>Content-Type</TT
>.
				</P
>
			</LI
><LI
>&#13;				<P
>&#13;					<B
CLASS="command"
>CheckMP3Only</B
>
<A
NAME="AEN875"
></A
>
				</P
>

				<P
>&#13;					This value is equal to <TT
CLASS="literal"
>CheckMP3</TT
>, but if MP3 tag is not present,
processing on <TT
CLASS="literal"
>Content-Type</TT
> will not be taken.
				</P
>
			</LI
></OL
>
		</P
>










<P
>&#13;  Use optional <CODE
CLASS="option"
>subsection</CODE
>
parameter to specify server's checking behavior.  Subsection value must be
one of the following: <TT
CLASS="literal"
>nofollow</TT
>, <TT
CLASS="literal"
>page</TT
>, <TT
CLASS="literal"
>path</TT
>, 
<TT
CLASS="literal"
>site</TT
>, <TT
CLASS="literal"
>world</TT
> and has "path" value by
default. 
		<P
></P
><OL
TYPE="1"
><LI
>&#13;				<P
>&#13;					<TT
CLASS="literal"
>path</TT
> subsection</P
>
				<P
>When indexer seeks for a
"Server" command corresponding to an URL it checks that the discovered
URL starts with URL given in Server command argument but without
trailing file name. For example, if <TT
CLASS="literal"
>Server path
http://localhost/path/to/index.html</TT
> is given, all URLs which
have <TT
CLASS="literal"
>http://localhost/path/to/</TT
> at the beginning
correspond to this Server command.</P
>

				<P
>The following commands have the same effect except that they insert different URLs into database:</P
>
				<P
>&#13;					<PRE
CLASS="programlisting"
>&#13;Server path http://localhost/path/to/index.html
Server path http://localhost/path/to/index
Server path http://localhost/path/to/index.cgi?q=bla
Server path http://localhost/path/to/index?q=bla
</PRE
>
				</P
>
			</LI
><LI
>&#13;				<P
>&#13;					<TT
CLASS="literal"
>site</TT
> subsection</P
>
				<P
>indexer checks that the
discovered URL have the same hostname with URL given in Server
command. For example, <TT
CLASS="literal"
>Server site
http://localhost/path/to/a.html</TT
> will allow to index whole
<TT
CLASS="literal"
>http://localhost/</TT
> server. </P
>

			</LI
><LI
>&#13;				<P
>&#13;					<TT
CLASS="literal"
>world</TT
> subsection</P
>
				<P
>If world subsection is specified
in Server command, it has the same effect that URL is considered to
match this Server command. See explanation below.</P
>

			</LI
><LI
>&#13;				<P
>&#13;					<TT
CLASS="literal"
>page</TT
> subsection</P
>
				<P
>This subsection describes the only one URL given in Server argument.</P
>
			</LI
><LI
>&#13;<P
><TT
CLASS="literal"
>nofollow</TT
> subsection</P
>
<P
>Skip links following for URL that match the pattern.</P
>
</LI
><LI
>&#13;				<P
>subsection in <TT
CLASS="literal"
>news://</TT
> schema</P
>
				<P
>Subsection is always considered
as "site" for news:// URL schema. This is because news:// schema has
no nested paths like ftp:// or http://  Use  <TT
CLASS="literal"
>Server
news://news.server.com/</TT
> to index whole news server or for
example <TT
CLASS="literal"
>Server news://news.server.com/udm</TT
> to index
all messages from "udm" hierarchy.</P
>

			</LI
></OL
>
</P
>


<P
>&#13;Optional parameter <TT
CLASS="literal"
>CaseType</TT
> is specify the case sensivity for string comparison, it can take one of follow value: 
<TT
CLASS="literal"
>case</TT
> - case insensitive comparison, or <TT
CLASS="literal"
>nocase</TT
> - case sensitive comparison.
</P
>


		<P
>&#13;Optional parameter <TT
CLASS="literal"
>CmpType</TT
> is specify the type of comparison and can take two value:
<TT
CLASS="literal"
>Regex</TT
> and <TT
CLASS="literal"
>String</TT
>.
<TT
CLASS="literal"
>String</TT
> wildcards is default
match type. You can use ? and * signs in URLMask parameters, they
means "one character" and "any number of characters" respectively. For
example, if you want to index all HTTP sites in .ru domain, use this
command:<PRE
CLASS="programlisting"
>Realm http://*.ru/*</PRE
>
		</P
>
		
		<P
>Regex comparison type takes a regular expression
as it's argument. Activate regex comparison type using <CODE
CLASS="option"
>Regex</CODE
>
keyword. For example, you can describe everything in .ru domain using
regex comparison type: <PRE
CLASS="programlisting"
>Realm Regex ^http://.*\.ru/</PRE
>
		</P
>

		
		<P
>Optional parameter <TT
CLASS="literal"
>MatchType</TT
> means match type. There
are <TT
CLASS="literal"
>Match</TT
> and <TT
CLASS="literal"
>NoMatch</TT
> possible values with <TT
CLASS="literal"
>Match</TT
> as
default. <TT
CLASS="literal"
>Realm NoMatch</TT
> has reverse effect. It means
that URL that does not match given <CODE
CLASS="option"
>pattern</CODE
> will correspond to this
<B
CLASS="command"
>Realm</B
> command. For example, use this command to index everything
without .com domain:<PRE
CLASS="programlisting"
>Realm NoMatch http://*.com/*</PRE
>
		</P
>
		
		<P
>Optional <CODE
CLASS="option"
>alias</CODE
> argument allows providing very
complicated URL rewrite more powerful than other aliasing
mechanism. Take a look  for <CODE
CLASS="option"
>alias</CODE
> argument usage
explanation. <CODE
CLASS="option"
>Alias</CODE
> works only with <CODE
CLASS="option"
>Regex</CODE
> comparison type and has no
effect with <CODE
CLASS="option"
>String</CODE
> type.</P
>








	<DIV
CLASS="sect2"
><H2
CLASS="sect2"
><A
NAME="follow-server"
>3.6.1. <B
CLASS="command"
>Server</B
> command</A
></H2
>
		
<A
NAME="AEN955"
></A
>

		<P
>This is the main command of the
<TT
CLASS="filename"
>indexer.conf</TT
> file. It is used to add servers or
their parts to be indexed. 

This command also says indexer to insert given URL into database at startup.</P
>


		<P
>E.g. command <TT
CLASS="literal"
>Server
http://localhost/</TT
>  allows  to index whole
<TT
CLASS="literal"
>http://localhost/</TT
> server. It also makes indexer
insert given URL into database at startup.  You can also specify some
path to index server subsection: <TT
CLASS="literal"
>Server
http://localhost/subsection/</TT
>. It also says indexer to insert
given URL at startup.</P
>

		<DIV
CLASS="note"
><BLOCKQUOTE
CLASS="note"
>&#13;			<P
><B
>Note: </B
>You can suppress indexer behavior to add
URL given in Server command by using -q indexer command line
argument. It is useful when you have hundreds or thousands Server
commands and their URLs are already in database. This allows having
more quick indexer startup.</P
>

		</BLOCKQUOTE
></DIV
>
	</DIV
>


	<DIV
CLASS="sect2"
><H2
CLASS="sect2"
><A
NAME="follow-realm"
>3.6.2. <B
CLASS="command"
>Realm</B
> command</A
></H2
>
		
<A
NAME="AEN969"
></A
>

		<P
>Realm command is a more powerful means of describing web area to be indexed.

It works almost like <B
CLASS="command"
>Server</B
> command but takes
a regular expression or string wildcards as it's <CODE
CLASS="option"
>pattern</CODE
> parameter and
do not insert any URL into database for indexing.
		</P
>



	</DIV
>


	<DIV
CLASS="sect2"
><H2
CLASS="sect2"
><A
NAME="follow-subnet"
>3.6.3. <B
CLASS="command"
>Subnet</B
> command</A
></H2
>
		
<A
NAME="AEN978"
></A
>

		<P
>Subnet command is another way to describe web area to be indexed.

It works almost like <B
CLASS="command"
>Server</B
> command but takes
a string wildcards or network specified in CIDR presentation format as it's <CODE
CLASS="option"
>pattern</CODE
> argument which is compared against IP
address instead of URL. In case of string wilcards formant, argument may have * and ? signs, they means
"one character" and "any number of characters" respectively. For
example, if you want to index all HTTP sites in your local subnet,
use this command:<PRE
CLASS="programlisting"
>Subnet 192.168.*.*</PRE
>
In case of network specified in CIDR presentation format, you may specify subnet in forms:  a.b.c.d/m, a.b.c, a.b, a
<PRE
CLASS="programlisting"
>Subnet 1291.168.10.0/24</PRE
>
		</P
>
		<P
>You may use "NoMatch" optional argument. For
example, if you want to index everything without
<TT
CLASS="literal"
>195.x.x.x</TT
> subnet, use:<PRE
CLASS="programlisting"
>Subnet NoMatch 195.*.*.*</PRE
>
		</P
>
	</DIV
>

	<DIV
CLASS="sect2"
><H2
CLASS="sect2"
><A
NAME="follow-difparam"
>3.6.4. Using different parameter for server and it's subsections</A
></H2
>
		
		<P
>Indexer seeks for "Server" and "Realm" commands
in order of their appearance. Thus if you want to give different
parameters to e.g. whole server and its subsection you should add
subsection line before whole server's. Imagine that you have server
subdirectory which contains news articles. Surely those articles are
to be reindexed more often than the rest of the server. The following
combination may be useful in such cases:</P
>

		<P
>&#13;			<PRE
CLASS="programlisting"
>&#13;# Add subsection
Period 200000
Server http://servername/news/

# Add server
Period 600000
Server http://servername/
</PRE
>
		</P
>
		<P
>These commands give different reindexing period
for <TT
CLASS="filename"
>/news/</TT
> subdirectory comparing with the period
of server as a whole. indexer will choose the first "Server" record
for the <TT
CLASS="filename"
>http://servername/news/page1.html</TT
> as far
as it matches and was given first.</P
>

	</DIV
>
	<DIV
CLASS="sect2"
><H2
CLASS="sect2"
><A
NAME="follow-default"
>3.6.5. Default <B
CLASS="command"
>indexer</B
> behavior</A
></H2
>
		
		<P
>The default behavior of indexer is to follow
through links having correspondent Server/Realm command in the
<TT
CLASS="filename"
>indexer.conf</TT
> file. It also jumps between servers
if both of them are present in <TT
CLASS="filename"
>indexer.conf</TT
>
either directly in Server command or indirectly in Realm command. For
example, there are two Server commands:</P
>

		<P
>&#13;			<PRE
CLASS="programlisting"
>&#13;Server http://www/
Server http://web/
</PRE
>
		</P
>
		<P
>When indexing
<TT
CLASS="filename"
>http://www/page1.html</TT
> indexer WILL follow the
link <TT
CLASS="filename"
>http://web/page2.html</TT
> if the last one has
been found. Note that these pages are on different servers, but BOTH
of them have correspondent Server record.</P
>

		<P
>If one of the Server command is deleted, indexer
will remove all expired URLs from this server during next
reindexing.</P
>

	</DIV
>

	<DIV
CLASS="sect2"
><H2
CLASS="sect2"
><A
NAME="follow-f"
>3.6.6. Using <KBD
CLASS="userinput"
>indexer -f &#60;filename&#62;</KBD
></A
></H2
>
		
		<P
>The third scheme is very useful for
<TT
CLASS="literal"
>indexer -i -f url.txt</TT
> running. You may maintain
required servers in the <TT
CLASS="filename"
>url.txt</TT
>. When new URL is
added into <TT
CLASS="filename"
>url.txt</TT
> indexer will index the server
of this URL during next startup. </P
>


	</DIV
>


<DIV
CLASS="sect2"
><H2
CLASS="sect2"
><A
NAME="DB_cmd"
>3.6.7. <B
CLASS="command"
>ServerDB, RealmDB, SubnetDB and URLDB</B
> commands</A
></H2
>

<A
NAME="AEN1019"
></A
>
<A
NAME="AEN1022"
></A
>
<A
NAME="AEN1025"
></A
>
<A
NAME="AEN1028"
></A
>
<PRE
CLASS="programlisting"
>&#13;URLDB pgsql://foo:bar@localhost/portal/links?field=url
</PRE
>
<P
>&#13;These commands are equal to <B
CLASS="command"
>Server</B
>, <B
CLASS="command"
>Realm</B
>, <B
CLASS="command"
>Subnet</B
> and 
<B
CLASS="command"
>URL</B
> commands respectively, but takes arguments from field of SQL-table specified. 
In example above, URLs are takes from database <CODE
CLASS="option"
>portal</CODE
>, SQL-table <CODE
CLASS="option"
>links</CODE
>
and filed <CODE
CLASS="option"
>url</CODE
>.
</P
>
</DIV
>

<DIV
CLASS="sect2"
><H2
CLASS="sect2"
><A
NAME="URL_cmd"
>3.6.8. <B
CLASS="command"
>URL</B
> command</A
></H2
>

<A
NAME="AEN1043"
></A
>
<PRE
CLASS="programlisting"
>&#13;URL http://localhost/path/to/page.html
</PRE
>
<P
>&#13;This command inserts given <CODE
CLASS="option"
>URL</CODE
> into database. This is usefull to add
several entry points to one server. Has no effect if an URL is already
in the database. 
</P
>
</DIV
>


</DIV
><DIV
CLASS="NAVFOOTER"
><HR
ALIGN="LEFT"
WIDTH="100%"><TABLE
SUMMARY="Footer navigation table"
WIDTH="100%"
BORDER="0"
CELLPADDING="0"
CELLSPACING="0"
><TR
><TD
WIDTH="33%"
ALIGN="left"
VALIGN="top"
><A
HREF="dpsearch-clones.en.html"
ACCESSKEY="P"
>Prev</A
></TD
><TD
WIDTH="34%"
ALIGN="center"
VALIGN="top"
><A
HREF="index.en.html"
ACCESSKEY="H"
>Home</A
></TD
><TD
WIDTH="33%"
ALIGN="right"
VALIGN="top"
><A
HREF="dpsearch-aliases.en.html"
ACCESSKEY="N"
>Next</A
></TD
></TR
><TR
><TD
WIDTH="33%"
ALIGN="left"
VALIGN="top"
>Clones</TD
><TD
WIDTH="34%"
ALIGN="center"
VALIGN="top"
><A
HREF="dpsearch-indexing.en.html"
ACCESSKEY="U"
>Up</A
></TD
><TD
WIDTH="33%"
ALIGN="right"
VALIGN="top"
>Aliases</TD
></TR
></TABLE
></DIV
><!--#include virtual="body-after.html"--></BODY
></HTML
>